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Abstract 

Background: Women diagnosed with early stage (I or II) breast cancer face a highly challenging decision - whether or 
not to undergo adjuvant chemotherapy. We developed a decision quality instrument for chemotherapy for early stage 
breast cancer and sought to evaluate its performance. 

Methods: Cross-sectional, mailed survey of recent breast cancer survivors, providers, and healthy controls and a retest 
survey of survivors. The decision quality instrument includes questions on knowledge and personal goals. It results in a 
knowledge score and concordance score, which reflects the percentage of patients who received treatments that 
match their goals. Hypotheses related to acceptability, feasibility, validity, and reliability of the survey instrument were 
examined. 

Results: Responses were received from 352 patients, 89 providers and 35 healthy controls. The decision quality 
instrument was feasible to implement with few missing data. The knowledge scores had good retest reliability 
(intraclass correlation coefficient (ICC) =0.75). Knowledge scores discriminated between providers and patients (mean 
difference 31.1%, 95% CI 26.9, 35.3) and between patients and healthy controls (mean difference 1 1.2, 95% CI 5.4, 17.1). 
Most providers reported that the knowledge items covered essential content. Two of the five goal items had a ceiling 
effect, and one goal had low content validity. The goal items had moderate retest reliability (ICC's 0.57 to 0.78). In the 
multivariable model of treatment, none of the patient goals was associated with receipt of chemotherapy. Age and 
hormone receptor status were the only variables independently associated with chemotherapy. Most patients (77.6%) 
had treatment concordant with that predicted by the model. Patients who had concordant treatment had similar 
levels of confidence and regret as those who did not. 

Conclusions: The Decision Quality Instrument is a reliable and valid measure of patient knowledge about 
chemotherapy, but its ability to measure concordance with patient goals is limited. In this sample, patient goals were 
not associated with treatment, and most patients reported they were not asked their preference, suggesting that goals 
were not adequately considered in decision making. 

Keywords: Decision, Regret, Breast cancer. Shared decision making. Quality of care. Decision quality. Chemotherapy, 
Adjuvant therapy 
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Background 

Women diagnosed with early stage (I or II) breast cancer 
face a highly challenging decision - whether or not to 
undergo adjuvant chemotherapy. Chemotherapy can 
provide gains in survival after breast cancer, but it also 
poses risks of serious toxicity. Because of this tradeoff 
between survival gains and toxicity risks, clinical guide- 
lines leave the decision about chemotherapy open to the 
patient and provider, for most patients with stage I dis- 
ease [1,2]. For Stage I patients, five year survival is quite 
high, and chemotherapy has a small absolute impact on 
10-year survival [3-5]. Thus, the decision about chemo- 
therapy for such patients is a "preference sensitive" one, 
for which medical evidence is necessary but not suffi- 
cient, and the right choice depends heavily on the pa- 
tient's personal preferences [6]. For patients with stage II 
disease, guidelines clearly recommend chemotherapy [1]. 
For those patients, the decision about chemotherapy 
should be an informed one, for which the patient fully 
understands the risks and benefits of the treatment. 

Breast cancer patients report challenges and dissatis- 
faction in making decisions about their treatment [7], 
and some patients have significant gaps in knowledge or 
unrealistic expectations of the benefits of systemic ther- 
apy [8-12]. Patients tend to rely heavily on provider rec- 
ommendations about chemotherapy [13], but providers 
do not always know or elicit patients' preferences [14] 
and they may convey information in ways that patients 
do not understand [15]. Overall, the evidence suggests 
that early-stage patients may not be meeting acceptable 
standards for informed consent when making decisions 
with their providers about adjuvant chemotherapy. 

Reducing the rate of uninformed or preference- 
insensitive chemotherapy decisions will require better 
measures of decision quality, due to a lack of reliable 
measures in these domains. A recent international con- 
sensus process has defined decision quality as the extent 
to which patients are informed and receive treatments 
that match their goals [6]. Valid measures of patient 
knowledge and preferences for breast cancer chemother- 
apy decisions, however, are lacking. Most studies of pa- 
tient preferences have focused on the minimum survival 
gain that patients require to accept chemotherapy and 
have not examined the full set of goals and concerns that 
patients express, including risks of complications and 
side effects [8,16-19]. Studies of knowledge have relied 
on ad hoc measures [8]. 

To address the need for better assessment of quality in 
chemotherapy decisions, we developed a survey instru- 
ment to measure patient knowledge and preferences 
about chemotherapy for early stage breast cancer, and 
evaluated its psychometric properties in a field test in 
breast cancer survivors. The instrument results in two 
scores, a knowledge score, indicating the extent to which 



patients are informed, and a concordance score, indicat- 
ing the percentage of patients who received treatments 
that matched their goals. To compare the instrument's 
performance in groups with different levels of know- 
ledge, we also surveyed breast cancer providers and 
healthy volunteers. The purpose of the field test was to 
evaluate feasibility, validity, and reliability of a new in- 
strument that has never been used with breast cancer 
patients. Given the stress and vulnerability of newly di- 
agnosed patients, we decided to conduct the field test 
with patients who had recovered from treatment. This 
approach would provide sufficient data to determine 
whether the questions were performing as intended, 
identify problematic wording or other issues, and allow 
some freedom to include more items than might be rea- 
sonably asked of newly diagnosed patients. 

Methods 

Initial instrument development 

The approach to measuring decision quality draws from 
normative and behavioral decision making theories and 
is based on the conceptual framework of shared decision 
making, as described by Mulley [20] and extended by 
Mulley and Sepucha [21]. Decision quality is defined as 
the extent to which patients are informed and receive 
treatments that match their goals [6]. Thus the two pri- 
mary domains of the instruments are knowledge of key 
facts and incorporation of patient goals and concerns 
into the decision. 

Development of the Decision Quality Instrument 
(BCS-DQI©) followed best practices of survey research 
methods [22-24]. We derived the initial set of facts and 
goals from a thorough review of the clinical literature 
and from focus groups of breast cancer patients. The 
candidate facts and goals were then rated by conveni- 
ence samples of patients (n = 17) and a multidisciplinary 
group of providers (n = 27) to evaluate accuracy, import- 
ance and completeness [25,26]. Experts in survey re- 
search drafted items to cover the key content and 
conducted cognitive interviews with patients (n = 6) in 
order to ensure comprehension and acceptability. Of the 
patients who participated in cognitive interviews, two 
had Stage I disease, 3 had Stage II, and one had un- 
known stage. Time since diagnosis ranged from 6 months 
to 5 years. 

Study design 

The instrument was tested in a cross-sectional survey of 
breast cancer patients and providers at four National 
Cancer Institute-designated cancer centers in 2008, 
along with healthy volunteers from one of the sites. The 
four cancer centers were academically affiliated and lo- 
cated in urban areas in the Northeast, Southeast, and 
West Coast of the United States. The study was 
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approved by the Institutional Review Boards at the Dana 
Farber Cancer Institute, Massachusetts General Hospital, 
University of California San Francisco, and University of 
North Carolina Chapel Hill. 

Samples and procedures 

Eligible patients were adult women, diagnosed in 2005 
or 2006 with early stage breast cancer, as reported in the 
cancer registry at each site. Exclusion criteria were a 
diagnosis prior to 2005, Stage 3 or higher, ductal carcin- 
oma in situ only, bilateral breast cancer, receipt of neo- 
adjuvant(before surgery) chemotherapy, and inability to 
read or speak English. The patients' treating physicians 
approved contact. 

Eligible patients received the study packet, including 
an informed consent form, in the mail. Respondents 
returned their signed informed consent form, signed 
authorization to access the medical record, and com- 
pleted study questionnaire by mail. After two weeks, the 
study staff called those who did not opt out. After four 
weeks, non-respondents received a reminder mailing 
and follow up phone call. Respondents received a thank- 
you letter and small compensation (e.g., $8 book of 
stamps) for completing the survey. A subset of respon- 
dents across all sites received a second survey by mail 
approximately four weeks after responding, to assess 
test-retest reliability of the decision quality instrument. 

A multidisciplinary group of breast cancer providers 
(medical oncologists, radiation oncologists, surgical on- 
cologists, plastic surgeons, and general surgeons) was 
identified by the principal investigator at each site. 
Healthy volunteers who had no history of breast cancer 
and were not clinicians were recruited through employee 
email bulletins of Partners Healthcare in Massachusetts. 
Providers and healthy volunteer participants received the 
knowledge portion of the survey by mail. All potential 
participants received an email reminder after two weeks 
and a mailed reminder after four weeks. Providers re- 
ceived $50 cash or a gift card upon completion of the 
survey. Healthy volunteers received a book of stamps 
upon completion. 

Measures 

We collected data on demographics and treatment his- 
tory from each site's cancer registry and the patient's re- 
port on the survey. When patient self-report conflicted 
with the registry on a medical or treatment issue (e.g., 
stage of disease), we examined the medical record to re- 
solve the issue. The survey included the following 
measures: 

1. Breast Cancer Systemic Therapy Decision Quality 
Instrument (BCS-DQI ©, see Additional file 1). 
Items cover decisions about adjuvant chemotherapy 



and hormone therapy. Here we report on the 
chemotherapy items only. Findings regarding the 
hormone therapy items will be reported separately. 
The instrument has two primary domains: 

a. Knowledge: 8 multiple choice and 4 open-ended 
items about recurrence, survival, treatment pro- 
cesses, and risks associated with chemotherapy. 

b. Goals and concerns: 5 items rating on a scale 
from 0 (not at all important) to 10 (extremely 
important). 

2. Decision process: 4 items about the nature of the 
interaction with health care providers including 
discussion of chemotherapy, pros and cons, and 
patients' preferences. In addition, 1 item asked 
patients about who made the final decision about 
treatments (mainly the doctor, mainly me, both 
equally). 

3. Patient's preferred treatment: A single item asked, 
"Which treatment was your personal preference?" 
with 4 possible responses: chemotherapy only, 
hormone therapy only, both chemotherapy and 
hormone therapy, not have chemotherapy and not 
have hormone therapy. 

4. Treatment received: A single item asking "Did you 
have chemotherapy?" 

5. Informed: A single item asking "How informed did 
you feel about chemotherapy treatments?" with a 0 
to 10 rating scale where 0 (not informed at all) and 
10 (extremely well informed). 

Confidence: A single item asking "How confident were 
you that the decision about chemotherapy was the right 
one?" with a 0 to 10 rating scale where 0 (not confident 
at all) and 10 (extremely confident). This item has been 
used in large national studies of decision making. Those 
studies found that patients reporting a stronger treat- 
ment preference and more involvement in decision mak- 
ing had higher confidence [27,28]. 

6. Regret: A single item asking "If you had to do it 
again, would you have the same treatments?" with a 
5-point Likert scale. The response options were: def- 
initely have the same, probably have the same, not 
sure, probably not have the same, and definitely not 
have the same. This item has been used in national 
studies of decision making that found higher prefer- 
ence concordance was associated with lower regret 
[29]. 

Healthy Volunteer survey: Healthy volunteers com- 
pleted the knowledge questions and some demographic 
items. 

Provider survey: Providers completed the knowledge 
questions, reported how important each item was 
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(essential, important, not important), and rated how well 
the knowledge questions covered key content (extremely 
well, very well, somewhat well, not at all well). We did 
not ask providers to suggest additional knowledge items, 
since we had done that in prior stages of instrument de- 
velopment [26]. 

Analysis 

Sample evaluation 

We began by evaluating demographics and response 
rates of the three samples (patient, provider, healthy 
volunteer). 

Item retention and deletion 

We evaluated each item in terms of problems with for- 
mat, level of difficulty, redundancy, and floor and ceil- 
ing effects. A problem with format was defined as more 
than 5% of respondents having a problem, such as 
marking two answers when one was expected. A know- 
ledge item was identified as possibly too difficult if 
fewer than 50% of providers responded correctly, or too 
easy if more than 80% of healthy volunteers answered 
correctly. An item was identified as possibly redundant 
if inter-item correlation was greater than 0.8. A steering 
committee composed of six experts in survey research, 
decision sciences, and breast cancer care (including two 
of the authors KS and CL) reviewed these analyses and 
recommended items for revision or deletion. 

Knowledge score 

A total knowledge score for each patient was calculated, 
using the items that were retained after the item retention 
and deletion process. The knowledge score was defined 
as the percentage of knowledge questions answered cor- 
rectly. Missing responses to the knowledge items were 
considered incorrect, and any respondent who did not 
complete at least half of the knowledge items did not re- 
ceive a knowledge score. An aggregate total knowledge 
score (0-100%) was computed for each sample. 

Concordance score 

We limited the concordance analysis to Stage I patients 
since chemotherapy has modest absolute impact on their 
year survival [3-5], and guidelines recommend consider- 
ing (rather than definitely providing) chemotherapy, so 
their decisions are more preference-sensitive. The con- 
cordance analysis began by examining each goal to deter- 
mine whether or not it distinguished between those who 
did and did not have chemotherapy. Then we included 
the goals, along with age, and hormone receptor status 
(defined as estrogen and progesterone receptor positive 
or not) in a multivariable logistic regression model, with 
receipt of chemotherapy as the dependent variable. To 
take into account correlation of observations from the 



same hospital, a Generalized Estimating Equations (GEE) 
approach was used with study site as a clustering variable. 
A patient who had predicted probability >0.5 and re- 
ceived chemotherapy, or who had predicted probabil- 
ity <0.5 and did not receive chemotherapy, was 
defined as having care that "matched" her goals. A 
summary concordance score (0-100%) was calculated 
by dividing the number of patients who matched by 
the total number of patients. 

In addition to this concordance analysis in all Stage I 
patients, we performed an additional analysis in Stage I 
patients with hormone receptor-negative tumors, since 
guidelines provide stronger recommendations for chemo- 
therapy in these patients [1]. 

Acceptability and feasibility 

We examined acceptability using instrument completion 
time, response rates, and written comments. We only 
had an estimate of completion time because participants 
had reported completion time for the whole survey, 
which included scales other than the systemic therapy 
DQI. We evaluated feasibility using item non-response 
patterns. Any item with more than 5% missing responses 
was subject to revision. 

Assessment of reliability 

To examine retest reliability, we calculated the intra-class 
correlation coefficient for the total knowledge score and 
for the individual goals and concerns. The target was to 
exceed 0.7. Because the knowledge items were not based 
on a single underlying construct, we did not calculate 
Cronbach's alpha or evaluate internal consistency. 

Validity of knowledge score 

For discriminant validity of the knowledge items, we 
tested the hypothesis that the provider total knowledge 
score would be higher than the patient score, and the 
patient score would be higher than the healthy volunteer 
score (using ANOVA with planned comparison). For 
content validity, we evaluated whether or not at least 
70% of providers reported that the knowledge items cov- 
ered key content. 

Validity of concordance score 

This analysis was limited to Stage I patients. For con- 
struct validity of the concordance model, we tested 
whether or not patients who stated a preference for 
chemotherapy had a higher model-predicted probability 
of chemotherapy than those who were unsure, using a 
two-sample f-test. We examined the hypothesis that 
concordance would be associated with less regret using 
a Chi-squared test. We also tested the hypothesis that 
concordance would be associated with more confidence 
in decisions using a two-sample i-test. For content 
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validity of the goal items, we evaluated whether or not at 
least 20% of patients included each item in their top 
three most important goals. 

Results 

Sample characteristics 

This study was part of a larger study of breast cancer 
treatment decisions, in which some patients received the 
survey on systemic therapy, and others received a survey 
on breast reconstruction. In the overall study, 456 out of 
768 patients (59.4%) participated in the initial survey, 
and 125/137 (91.2%) in the retest survey. This paper in- 
cludes 352 who completed the systemic therapy initial 
survey and 89 systemic therapy retest surveys. Many of 
those patients (237 of 352, 67.3%) had Stage I disease 
and were included in the concordance analyses. Among 



the Stage I patients, 35.4% received chemotherapy, alone 
or in combination with hormone therapy (Table 1). The 
rate of chemotherapy did not vary by site, for Stage I or 
Stage II patients. The mean length of time since diagno- 
sis was 30.8 months (SD 9.8). The sample of 35 healthy 
volunteers was similar to the patient sample, in terms of 
race, education, marital status, and reported health status, 
but was younger on average (42 vs. 57 years, p <0.001) 
(Table 1). Most providers who received the survey com- 
pleted it (89 of 116, 77%). The providers' average age 
was 45 years (SD 9), most were female (65%), and they 
had been in practice an average of 15 years (SD 11). 
The provider sample was multidisciplinary, including 
medical oncologists (33%), nurses (29%), surgeons/sur- 
gical oncologists(17%), radiation oncologists (8%) and 
plastic surgeons 



Table 1 Characteristics of patients and healthy volunteers 



Patients 



Stage I 



Stage II 



Total 



Healthy 
volunteers 



All 


237 


(67.3) 


115 


(32.7) 


352 


(100.0) 


35 




Age mean (SD) 


57 


(11) 


57 


(12) 


57 


(11) 


424 


(11) 


Race, n (%) 


















White 


199 


(84.0) 


92 


(80.0) 


291 


(82.7) 


26 


(74.3) 


Black 


17 


(7.2) 


13 


(11.3) 


30 


(8.5) 


3 


(8.6) 


Hispanic 


7 


(3.0) 


2 


(1.7) 


9 


(2.6) 


1 


(2.9) 


Asian, Hawaiian, Pacific Islander 


7 


(3.0) 


4 


(3.5) 


11 


(3.1) 


3 


(8.6) 


Other/unknown 


7 


(3.0) 


4 


(3.5) 


11 


(3.1) 


2 


(5.7) 


Education, n (%) 


















HS or less 


29 


(12.2) 


13 


(11.3) 


42 


(1 1 .9) 


0 


(0.0) 


Some college 


51 


(21.5) 


36 


(31.3) 


87 


(24.7) 


8 


(22.9) 


College or more 


157 


(66.2) 


66 


(574) 


223 


(63.4) 


27 


(77.1) 


Hormone receptor status, n (%) 


















Positive 


198 


(83.5) 


96 


(83.5) 


294 


(83.5) 


na 




Negative 


39 


(16.5) 


19 


(16.5) 


58 


(16.5) 


na 




Local treatment, n (%) 


















Lumpectomy 


172 


(72.6) 


80 


(69.6) 


252 


(71.6) 


na 




IVIastectomy 


65 


(27.4) 


35 


(304) 


100 


(28.4) 


na 




Systemic therapy, n (%) 


















Chemotherapy only 


33 


(13.9) 


21 


(18.3) 


54 


(15.3) 


na 




Hormone therapy only 


124 


(52.3) 


22 


(19.1) 


146 


(41.5) 


na 




Both 


51 


(21.5) 


65 


(56.5) 


116 


(33.0) 


na 




No systemic therapy 


29 


(12.2) 


7 


(6.1) 


36 


(10.2) 


na 




Months since diagnosis 


















Median (25th, 75th) 


29 


(23,38) 


31 


(22,38) 


30 


(23,38) 


na 




Overall Health, n (%) 


















Excellent/very good 


159 


(67.1) 


70 


(60.9) 


229 


(65.1) 


32 


(91.4) 


Other 


78 


(32.9) 


45 


(39.1) 


123 


(34.9) 


3 


(8.6) 



Abbreviations: na not applicable; SD standard deviation. 
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Item retention and deletion 

Two knowledge items were deleted for being too easy - 
"Can breast cancer come back after chemotherapy?" 
(correct answer "yes") and "Mark whether or not some 
women have this side effect from chemotherapy: a. hair 
loss" (correct answer"yes"). We also recommended redu- 
cing the number of open-ended quantitative estimate 
questions (e.g., "If 100 women with breast cancer like 
yours had chemotherapy, about how many would be 

cancer free in ten years? write in number of people") 

from 4 to 2, because of feedback that patients had diffi- 
culty distinguishing among the four recurrence/survival 
scenarios, and because the questions did not add more 
information. We kept some of the open-ended questions 
because they were on critical topics, such as recurrence. 

We recommended removal of the response option "I 
am not sure" because patients used it so frequently (fre- 
quency ranging from 11.4 to 49.4%) (Table 2), and it ap- 
peared to result in underestimation of patient knowledge. 

Two of the goal items had a ceiling effect. For "Live as 
live as long as possible," 284 of 352 patients (79.5%) rated 
the importance as 10 out of 10. For "Lower chance of re- 
currence", 311 of 354 (88.4%) rated the importance as 10 
out of 10. We kept these items for the concordance ana- 
lyses because most patients included them in their top 
three issues. However, the steering committee recom- 
mended revision in future versions to improve their ability 
to discriminate between patients who had chemotherapy 
and those who did not. 

At least 20% of patients selected each of the goals, ex- 
cept one, in their top three. One goal, "Avoid the costs 
of having additional treatments", was deleted for lack of 
content validity - only 5.4% of respondents selected it 
in their top three, and its selection did not vary by 
treatment. Patients who did not undergo chemotherapy 
had a slightly different frequency of selecting goals in 
their top three, compared to those who did undergo 
chemotherapy. For example, fewer patients who did not 
have chemotherapy selected "live as long as possible" 
(84.1 for those who did not have chemotherapy versus 
92.4% for those who did, p = 0.021) and "lower the 
chance of having cancer come back" (85.7% vs. 98.2%, 
p = 0.001), in their top three goals. 

Acceptability and feasibility 

The estimated mean completion time for the DQI (in- 
cluding the knowledge items, goals and decision 
process items) was 7.2 minutes (range 2.3, 22.9 mi- 
nutes). The knowledge items had a mean of 2.5% miss- 
ing responses (range 1.4 to 4.5%). Six respondents 
(1.7%) did not answer enough knowledge items to gen- 
erate a total knowledge score. For the goal items, the 
missing responses were also infrequent, with a mean of 
2.8% (range 0.4 to 4.4%). 



Table 2 Selected knowledge items and responses (of 1 2 
total items) 

Question and possible Number with this response, n (%) 

responses (correct response in patients Providers Controls 
Italics)* 



N 



352 



35 



How much would waiting 4 weel<s to make a decision about 
chemotlierapy and liormone tlierapy affect chances of survival? 



A lot 

Somewhat 

A little or not at all 

I am not sure 

IVlissing 



36 
54 
215 
40 
7 



(10.2) 
(153) 
(61.1) 
(11.4) 
(2.0) 



0 
5 
81 
1 
1 



(0) 
(6) 
(91) 
(1) 
(1) 



6 
14 
13 
2 
0 



(17) 
(40) 
(37) 
(6) 
(0) 



Without chemotherapy or hormone therapy, about how many women 
with early stage breast cancer will eventually die of breast cancer? 



More than half 
About half 
Less than half 
I am not sure 
IVlissing 



52 
44 
105 
144 

7 



(14.8) 
(12.5) 
(29.8) 
(40.9) 
(2.0) 



5 
7 

70 
3 
4 



(6) 
(8) 
(79) 
(3) 
(4) 



(40) 
(20) 
(29) 
(11) 
(0) 



Without chemotherapy or hormone therapy, about how many women 
with early stage breast cancer will be cancer free in ten years? 



More than half 
About half 
Less than half 
I'm not sure 
Missing 



77 
44 
76 



(21.9) 
(12.5) 
(21.6) 
(41.5) 
(2.3) 



15 



(54) 
(17) 
(20) 
(4) 
(4) 



4 

3 
21 
7 
0 



(11) 
(9) 
(60) 
(20) 
(0) 



Out of 100 women who have chemotherapy, how many will have a 
serious side effect, such as another cancer or serious heart problem? 



Fewer than 5 


70 


(19.9) 


59 


(66) 


5 


(14) 


5 to 10 


63 


(17.9) 


15 


(17) 


12 


(34) 


1 1 to 20 


23 


(6.5) 


4 


(4) 


7 


(20) 


More than 20 


18 


(5.1) 


1 


(1) 


4 


(11) 


1 am not sure 


166 


(47.2) 


7 


(8) 


7 


(20) 


Missing 


10 


(2.8) 


3 


(3) 


0 


(0) 



If 100 women with early breast cancer did not have chemotherapy or 
hormone therapy, how many would be cancer free in 1 0 years? 
( write in number) 



Answer less than 60 


139 


(39.5) 


18 


(20) 


28 


(80) 


Answer within 60-90 range 


79 


(22.4) 


31 


(35) 


1 


(3) 


Answer greater than 90 


13 


(3.7) 


0 


(0) 


0 


(0) 


Not sure 


115 


(32.7) 


15 


(17) 


6 


(17) 


Missing 


6 


(1.7) 


25 


(28) 


0 


(0) 


Total knowledge score, mean (SD) 


40 


(21.0) 


71 


(17) 


29 


(16) 



*Some questions have been shortened to fit this table. 



Reliability 

The total knowledge score had an intra-class correlation 
coefficient (ICC) of 0.75. 
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The goal items had coefficients ranging from 0.57 to 
0.78. The individual coefficients were as follows: "avoid 
serious risks of chemotherapy such as heart problems 
and other cancers" ICC = 0.58, "have a treatment that 
was convenient to complete" ICC = 0.66, "avoid having 
to take medicine for several years" ICC = 0.67, "avoid the 
costs of having additional treatments" ICC = 0.69, "live 
as long as possible" ICC = 0.69, "lower the chance of 
having cancer come back" ICC = 0.78. 

Validity of l<nowledge score 

The average patient knowledge score, among the 346 
(98.3%) patients who completed at least half of the 
knowledge questions, was 40.1 (SD 20.9). The average 
provider knowledge score (71.2, SD 17.0) was higher 
than the patient score (mean difference 31.1, 95% confi- 
dence interval 26.9, 35.3). The average healthy volunteer 
knowledge score (28.8, SD 16.0) was lower than the pa- 
tient score (mean difference 11.2, 95% confidence inter- 
val 5.4, 17.1). Most providers (62%) reported that the 
knowledge items covered key content extremely or very 
well. An additional 33% reported that the items covered 
key content somewhat well. In the written comments, 
providers did not note any specific missing content. 

Validity of concordance score 

These analyses were limited to Stage I patients (n = 232). 
On multivariable analysis, none of the patient goals was 
meaningfully associated with receipt of chemotherapy. 
Age and hormone receptor status were associated with 
chemotherapy in the multivariable model, with large odds 
ratios (Table 3). On bivariable analysis, several goals were 
associated with chemotherapy treatment. For "live as long 
as possible", the mean importance score was 9.87 in pa- 
tients who had chemotherapy and 9.64 in patients who 
did not (p = 0.008). For "lower the chance of having can- 
cer come back", the mean importance score was 9.93 in 
patients who had chemotherapy and 9.77 in those who 
did not (p = 0.02). For "avoid serious risks of chemother- 
apy such as heart problems and other cancers", the mean 
importance score was 7.01 in patients who had chemo- 
therapy and 7.76 in those who did not (p = 0.076). 



In the analysis limited to patients with Stage I and hor- 
mone receptor positive tumors, no goal was associated 
with having chemotherapy. Only age was associated with 
chemotherapy (odds ratio 0.478, 95% confidence interval 
0.332, 0.687). 

With the multivariable model of treatment, which in- 
cludes age, hormone receptor status, and goals, 77.6% of 
patients had treatment that was concordant with the 
model's prediction (Table 4). Patients who stated a pref- 
erence for chemotherapy had higher model-predicted 
probability of chemotherapy (0.52), compared to those 
who stated a preference for no chemotherapy or were 
unsure (0.28, p < 0.0001). Having concordance was not 
associated with less decisional regret (11.7% for concord- 
ant versus 15.4% for non-concordant, p = 0.48) or with 
more confidence in decisions (8.4 for concordant versus 
8.2 for non-concordant, p = 0.67). 

Decision process 

Patients' reports about their involvement in the decision 
process were variable (Table 5). A minority of patients 
(42.9%) reported that their doctor asked their preference. 
When asked who made the final decision, 45.7% re- 
ported "mainly me", 22.7% reported "mainly the doctor", 
and 29.5% reported "both equally". 

Discussion 

The Breast Cancer Systemic Therapy Decision Quality 
Instrument is designed to measure the degree to which 
patients' chemotherapy decisions are informed and 
concordant with personal goals. The knowledge score 
was reliable, and it discriminated among providers, pa- 
tients, and healthy volunteers. The goal items had 
modest retest reliability and strong content validity. 
Patients were able to complete the instrument quickly 
with few problems. 

Concordance 

We were somewhat surprised to find that none of the 
patients' goals was meaningfully associated with treat- 
ment in the concordance model. Although one goal 
("Have a treatment that was convenient to complete") 



Table 3 Multivariable logistic regression model of characteristics associated with chemotherapy in Stage 1 


1 patients 


Characteristic 


Odds ratio 


95% CI 


Negative hormone receptor status 


14.3 


5.51, 37.17 


Age (increment of 10) 


0.46 


0.36 0.59 


Live as long as possible 


1.45 


0.78, 2.66 


Have a treatment that was convenient to complete 


1.03 


1.01, 1.04 


Lower the change of having cancer come bacl< 


1.78 


0.76, 4.17 


Avoid the costs of having additional treatments 


1.01 


0.94, 1 .09 


Avoid serious risks of chemotherapy 


0.93 


0.86, 1.01 



Lee et at. BMC Medical Informatics and Decision Making 2014, 14:73 
http://www.biomedcentral.com/1472-6947/14/73 



Page 8 of 1 1 



Table 4 Concordance between predicted and actual 
treatment for Stage I patients 

Actual chemotherapy treatment Total 

Yes No 

Predicted chemotherapy treatment, n (%) 

Yes, n(%) 43 (18.5) 11 (4.7) 54 

No, n(%) 41 (17.7) 137 (59.1) 178 

Total*, n(%) 84 148 232 

% of the total 232 patients. 

*5 patients were not included in the analysis due to missing responses for all 
goal items. 

was statistically significant in the multivariable model, 
the odds ratio was so close to 1 (1.03) that we did not 
consider it clinically meaningful. One clinical factor, 
namely hormone receptor status, was the strongest pre- 
dictor of receipt of chemotherapy. Hormone receptor 
status, as well as other clinical and patient features, such 
as age and tumor stage, should, in fact, guide care deci- 
sions. Some clinicians might even argue that providers 
should recommend chemotherapy to their patients with 
hormone receptor negative disease. However, for hor- 
mone receptor positive patients, it is reasonable for pa- 
tients' goals to play a role in determining whether or not 
to undergo chemotherapy. When we restricted the con- 
cordance analyses to stage I, hormone receptor positive 
participants, we still found that only age was associated 
with receipt of chemotherapy. None of the goals ap- 
peared to have any impact on receipt of chemotherapy. 



even for hormone receptor positive patients. One inter- 
pretation of these results is that providers use clinical 
features to guide treatment choices, with little regard for 
patients' goals and preferences. 

Another possible interpretation is that the goal items 
in this instrument may not have captured aspects that 
were driving patients' decisions and as a result, the 
model may not reflect the tailoring of treatments to pa- 
tients' goals. Also, the responses to the goal items had 
little variability, limiting our ability to discriminate 
among the treatment options. Although the differences 
in the mean responses by treatment were statistically 
significant, the actual magnitude of the differences was 
quite small and would be difficult to use to guide clinical 
decisions. Future studies could explore alternative item 
and response formats for eliciting patients' goals to in- 
crease variability. 

A limitation of the concordance analysis is that a few 
of the Stage I patients may have met guideline criteria 
for chemotherapy (Her2 neu positive tumor larger than 
1 centimeter, triple negative tumor larger than 1 centi- 
meter, or high 21-gene assay score) [1,2]. We did not 
collect clinical data at that level of detail, however, so we 
could not remove those patients from the concordance 
analysis. 

Decision process 

Our data on the decision process suggest that providers 
may not be taking patients' goals and preferences into 



Table 5 Decision process by stage 



Decision process question^ n (%) Stage I Stage II Total 



Did any doctor tell you that having or not having chemotherapy was a choice for you? 














Yes 


159 


(71.3) 


80 


(69.6) 


249 


(70.7) 


No 


57 


(24.1) 


30 


(26.1) 


87 


(24.7) 


1 am not sure/missing 


11 


(5.1) 


5 


(4.3) 


16 


(4.8) 


How much did doctors discuss the reasons to have chemotherapy? 














A lot/some 


139 


(58.6) 


97 


(84.3) 


236 


(67.0) 


A little/not at al 


80 


(33.8) 


13 


(11.3) 


93 


(26.4) 


1 am not sure/missing 


18 


(7.6) 


5 


(4.3) 


23 


(6.5) 


How much did doctors discuss reasons not to have chemotherapy? 














A lot/some 


119 


(50.2) 


44 


(38.3) 


163 


(46.3) 


A little/not at al 


105 


(44.3) 


61 


(53.0) 


166 


(47.2) 


1 am not sure/missing 


13 


(5.5) 


10 


(8.6) 


23 


(6.5) 


Did any doctor ask you whether you preferred to have chemotherapy? 














Yes 


105 


(44.3) 


46 


(40.0) 


151 


(42.9) 


No 


103 


(43.5) 


56 


(48.7) 


159 


(45.2) 


1 am not sure/missing 


29 


(12.6) 


13 


(11.3) 


42 


(12.2) 


Mean decision process score (SD)** 


2.2 


(1.4) 


2.3 


(1.2) 


2.3 


(1.3) 



*Questions have been shortened to fit this table. 



^Score reflects 1 point for each question answered with yes, a lot, or some (range 0 to 4). 
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account. Less than half of the patients reported that 
their provider asked about their preferences for chemo- 
therapy. Discussing patients' goals and preferences is a 
core element of shared decision making and is important 
for patient-centered care. Patients and providers may 
not be viewing chemotherapy as a preference-sensitive 
decision for stage 1 patients, but instead considering it 
as effective care. Treatment guidelines, however, clearly 
state that use of chemotherapy for most patients with 
stage I disease is dependent upon patients' preferences 
[1]. Additional studies may be warranted to understand 
factors driving providers' recommendations for use of 
chemotherapy, including financial and organizational in- 
centives [30]. 

Knowledge 

The results provide evidence of the reliability and valid- 
ity of the knowledge score. Few other validated measures 
of knowledge about chemotherapy are available. A 2003 
randomized controlled trial of a decision aid about 
chemotherapy used a 25-item questionnaire that in- 
cluded true-false items about risk of breast cancer recur- 
rence and risks and benefits of chemotherapy [31]. The 
questionnaire was based on those used in prior studies 
of surgery and radiation decisions [32,33], but its psy- 
chometric properties have not been reported. Other ad 
hoc knowledge measures have focused on the accuracy 
of patients' estimates of their personal risk of breast can- 
cer recurrence or their estimate of the survival benefit of 
chemotherapy [17-19,34]. 

Patient knowledge about key facts related to chemo- 
therapy was quite low in our population, with average 
overall knowledge of 40.1%. Knowledge about the nat- 
ural history of early stage breast cancer was particularly 
low, with only 32.6% of patients answering correctly. 
Most patients who answered incorrectly overestimated 
mortality. Knowledge about survival with chemotherapy 
was low, with 27% of patients answering correctly. Most 
patients who answered incorrectly overestimated the 
benefit of chemotherapy on survival. 

Prior research suggests that patients tend to overesti- 
mate baseline recurrence, mortality rates, and the benefit 
of chemotherapy; while providers emphasize relative bene- 
fit of chemotherapy in terms of impact on 5 or 10 year re- 
currence [10-12]. This relative benefit is often in the range 
of 30%, which sounds very high to both patients and pro- 
viders. When providers educate patients about the abso- 
lute benefit of chemotherapy on ten-year mortality (which 
can range from 2-11%), both parties appear to recognize 
that the decision is preference-sensitive, and fewer pa- 
tients elect chemotherapy [9,10,17]. 

The study had some important limitations. Patients 
were an average of 2.6 years (SD 9.8) out from treat- 
ment, so they may have forgotten some information they 



knew at the time of decision making. Their responses to 
questions about goals may have been subject to recall 
bias, with a tendency to favor goals that were more con- 
sistent with the treatment they had. The study popula- 
tion was primarily white, educated, and higher-income 
than average, so the performance of the instrument in 
other populations remains unclear. This study was part 
of a larger study to validate other breast cancer decision 
quality measures, and the overall response rate for the 
larger study was lower among non-white patients [35]. 
We conducted the study at four academic comprehen- 
sive cancer centers, where select patients may be re- 
ferred, and practice styles may differ from that in the 
community, which may further limit the generalizability 
of our findings. Healthy volunteers were from one of the 
sites, so their responses to the knowledge questions may 
not be representative of healthy persons from other 
locations. 

Conclusions 

The Breast Cancer Systemic Therapy Decision Quality 
Instrument demonstrated strong reliability and validity 
as a measure of patient knowledge about chemotherapy 
for early stage breast cancer in this sample, but its ability 
to measure concordance was limited. In this population, 
no patient goal was associated with treatment, and most 
patients reported they were not asked their preference, 
suggesting that patients' goals may not have been ad- 
equately considered in treatment decision making about 
chemotherapy. Further testing will be important to 
understand the performance of the survey in newly diag- 
nosed patients who are actively making the decision, to 
evaluate alternative approaches to eliciting patients' 
goals, to examine the responsiveness of the knowledge 
score to interventions, such as patient decision aids, and 
to gather data to help set benchmarks for knowledge 
and concordance scores. 

Additional file 



Additional file 1: Breast Cancer Systemic Therapy Decision Quality 
Instrument. 
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